[PATCH 1/2] siggen: Drop copy_unihashes function

[PATCH 1/2] siggen: Drop copy_unihashes function

[Richard Purdie]

The code in OE-Core using this has been replaced with something more fit
for purpose. Drop these function calls as they were never a great idea in the
first place and cause IO slowdown for runqueue needing to sync the cache.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/siggen.py | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 92066da00c..89b70fb6a4 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -201,9 +201,6 @@ class SignatureGenerator(object):
     def save_unitaskhashes(self):
         return
 
-    def copy_unitaskhashes(self, targetdir):
-        return
-
     def set_setscene_tasks(self, setscene_tasks):
         return
 
@@ -418,9 +415,6 @@ class SignatureGeneratorBasic(SignatureGenerator):
     def save_unitaskhashes(self):
         self.unihash_cache.save(self.unitaskhashes)
 
-    def copy_unitaskhashes(self, targetdir):
-        self.unihash_cache.copyfile(targetdir)
-
     def dump_sigtask(self, mcfn, task, stampbase, runtime):
         tid = mcfn + ":" + task
         mc = bb.runqueue.mc_from_tid(mcfn)

[PATCH 2/2] runqueue: Avoid save_unitaskhashes

[Richard Purdie]

The save comes with an IO overhead which can slow down the rehash loop in bitbake
a lot. We only needed to do this when recipes were doing unihash cache copying. Now
they aren't doing that, drop this IO pain point.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/runqueue.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 93079a9776..3462ed4457 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2550,9 +2550,6 @@ class RunQueueExecute:
                     self.rqdata.runtaskentries[hashtid].unihash = unihash
                     bb.parse.siggen.set_unihash(hashtid, unihash)
                     toprocess.add(hashtid)
-                if torehash:
-                    # Need to save after set_unihash above
-                    bb.parse.siggen.save_unitaskhashes()
 
         # Work out all tasks which depend upon these
         total = set()

Re: [bitbake-devel] [PATCH 1/2] siggen: Drop copy_unihashes function

[Alexandre Belloni]

Hello Richard,


On 04/06/2024 15:28:02+0100, Richard Purdie wrote:
> The code in OE-Core using this has been replaced with something more fit
> for purpose. Drop these function calls as they were never a great idea in the
> first place and cause IO slowdown for runqueue needing to sync the cache.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/siggen.py | 6 ------
>  1 file changed, 6 deletions(-)
> 
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index 92066da00c..89b70fb6a4 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -201,9 +201,6 @@ class SignatureGenerator(object):
>      def save_unitaskhashes(self):
>          return
>  
> -    def copy_unitaskhashes(self, targetdir):
> -        return
> -
>      def set_setscene_tasks(self, setscene_tasks):
>          return
>  
> @@ -418,9 +415,6 @@ class SignatureGeneratorBasic(SignatureGenerator):
>      def save_unitaskhashes(self):
>          self.unihash_cache.save(self.unitaskhashes)
>  
> -    def copy_unitaskhashes(self, targetdir):
> -        self.unihash_cache.copyfile(targetdir)
> -

It breaks sstatetests.SStateBundles.test_minimal_bundle which uses copy_unitaskhashes

>      def dump_sigtask(self, mcfn, task, stampbase, runtime):
>          tid = mcfn + ":" + task
>          mc = bb.runqueue.mc_from_tid(mcfn)

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16316): https://lists.openembedded.org/g/bitbake-devel/message/16316
> Mute This Topic: https://lists.openembedded.org/mt/106483284/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [bitbake-devel] [PATCH 1/2] siggen: Drop copy_unihashes function

[Richard Purdie]

On Sat, 2024-06-08 at 12:16 +0200, Alexandre Belloni wrote:
> Hello Richard,
> 
> 
> On 04/06/2024 15:28:02+0100, Richard Purdie wrote:
> > The code in OE-Core using this has been replaced with something
> > more fit
> > for purpose. Drop these function calls as they were never a great
> > idea in the
> > first place and cause IO slowdown for runqueue needing to sync the
> > cache.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  lib/bb/siggen.py | 6 ------
> >  1 file changed, 6 deletions(-)
> > 
> > diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> > index 92066da00c..89b70fb6a4 100644
> > --- a/lib/bb/siggen.py
> > +++ b/lib/bb/siggen.py
> > @@ -201,9 +201,6 @@ class SignatureGenerator(object):
> >      def save_unitaskhashes(self):
> >          return
> >  
> > -    def copy_unitaskhashes(self, targetdir):
> > -        return
> > -
> >      def set_setscene_tasks(self, setscene_tasks):
> >          return
> >  
> > @@ -418,9 +415,6 @@ class
> > SignatureGeneratorBasic(SignatureGenerator):
> >      def save_unitaskhashes(self):
> >          self.unihash_cache.save(self.unitaskhashes)
> >  
> > -    def copy_unitaskhashes(self, targetdir):
> > -        self.unihash_cache.copyfile(targetdir)
> > -
> 
> It breaks sstatetests.SStateBundles.test_minimal_bundle which uses
> copy_unitaskhashes


Is that patch only in your -next branch? I didn't see this issue in my
own testing which makes me wonder...

Cheers,

Richard

Re: [bitbake-devel] [PATCH 1/2] siggen: Drop copy_unihashes function

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

I think it’s in my patch set for build replication that needs to be
reworked to not use the function and should be dropped until then.

Alex

On Sat 8. Jun 2024 at 12.18, Richard Purdie via lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:

> On Sat, 2024-06-08 at 12:16 +0200, Alexandre Belloni wrote:
> > Hello Richard,
> >
> >
> > On 04/06/2024 15:28:02+0100, Richard Purdie wrote:
> > > The code in OE-Core using this has been replaced with something
> > > more fit
> > > for purpose. Drop these function calls as they were never a great
> > > idea in the
> > > first place and cause IO slowdown for runqueue needing to sync the
> > > cache.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > >  lib/bb/siggen.py | 6 ------
> > >  1 file changed, 6 deletions(-)
> > >
> > > diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> > > index 92066da00c..89b70fb6a4 100644
> > > --- a/lib/bb/siggen.py
> > > +++ b/lib/bb/siggen.py
> > > @@ -201,9 +201,6 @@ class SignatureGenerator(object):
> > >      def save_unitaskhashes(self):
> > >          return
> > >
> > > -    def copy_unitaskhashes(self, targetdir):
> > > -        return
> > > -
> > >      def set_setscene_tasks(self, setscene_tasks):
> > >          return
> > >
> > > @@ -418,9 +415,6 @@ class
> > > SignatureGeneratorBasic(SignatureGenerator):
> > >      def save_unitaskhashes(self):
> > >          self.unihash_cache.save(self.unitaskhashes)
> > >
> > > -    def copy_unitaskhashes(self, targetdir):
> > > -        self.unihash_cache.copyfile(targetdir)
> > > -
> >
> > It breaks sstatetests.SStateBundles.test_minimal_bundle which uses
> > copy_unitaskhashes
>
>
> Is that patch only in your -next branch? I didn't see this issue in my
> own testing which makes me wonder...
>
> Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16331):
> https://lists.openembedded.org/g/bitbake-devel/message/16331
> Mute This Topic: https://lists.openembedded.org/mt/106483284/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3734 bytes --]

Re: [bitbake-devel] [PATCH 1/2] siggen: Drop copy_unihashes function

[Alexandre Belloni]

On 08/06/2024 11:18:48+0100, Richard Purdie wrote:
> On Sat, 2024-06-08 at 12:16 +0200, Alexandre Belloni wrote:
> > Hello Richard,
> > 
> > 
> > On 04/06/2024 15:28:02+0100, Richard Purdie wrote:
> > > The code in OE-Core using this has been replaced with something
> > > more fit
> > > for purpose. Drop these function calls as they were never a great
> > > idea in the
> > > first place and cause IO slowdown for runqueue needing to sync the
> > > cache.
> > > 
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > > �lib/bb/siggen.py | 6 ------
> > > �1 file changed, 6 deletions(-)
> > > 
> > > diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> > > index 92066da00c..89b70fb6a4 100644
> > > --- a/lib/bb/siggen.py
> > > +++ b/lib/bb/siggen.py
> > > @@ -201,9 +201,6 @@ class SignatureGenerator(object):
> > > ���� def save_unitaskhashes(self):
> > > �������� return
> > > �
> > > -��� def copy_unitaskhashes(self, targetdir):
> > > -������� return
> > > -
> > > ���� def set_setscene_tasks(self, setscene_tasks):
> > > �������� return
> > > �
> > > @@ -418,9 +415,6 @@ class
> > > SignatureGeneratorBasic(SignatureGenerator):
> > > ���� def save_unitaskhashes(self):
> > > �������� self.unihash_cache.save(self.unitaskhashes)
> > > �
> > > -��� def copy_unitaskhashes(self, targetdir):
> > > -������� self.unihash_cache.copyfile(targetdir)
> > > -
> > 
> > It breaks sstatetests.SStateBundles.test_minimal_bundle which uses
> > copy_unitaskhashes
> 
> 
> Is that patch only in your -next branch? I didn't see this issue in my
> own testing which makes me wonder...

You are right, git blame told me it was applied a month ag and I didn't
check further. I'll drop the patch.



-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com