* [PATCH 1/2] sstatetests: Use python function instead of bitbake-diffsigs script
@ 2017-08-04 18:57 leonardo.sandoval.gonzalez
2017-08-04 18:57 ` [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ leonardo.sandoval.gonzalez
0 siblings, 1 reply; 6+ messages in thread
From: leonardo.sandoval.gonzalez @ 2017-08-04 18:57 UTC (permalink / raw)
To: openembedded-core
From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Using a python function instead of launching a subprocess fasten the
diffsigs computation.
[YOCTO #11651]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
meta/lib/oeqa/selftest/cases/sstatetests.py | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
index 07a206824a..0b36027918 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -8,6 +8,8 @@ from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_test_layer
from oeqa.selftest.cases.sstate import SStateBase
from oeqa.core.decorator.oeid import OETestID
+import bb.siggen
+
class SStateTests(SStateBase):
# Test sstate files creation and their location
@@ -469,9 +471,11 @@ http_proxy = "http://example.com/"
for k in files1.keys() | files2.keys():
if k in files1 and k in files2:
print("%s differs:" % k)
- print(subprocess.check_output(("bitbake-diffsigs",
- self.topdir + "/tmp-sstatesamehash/stamps/" + k + "." + files1[k],
- self.topdir + "/tmp-sstatesamehash2/stamps/" + k + "." + files2[k])))
+ sigdatafile1 = self.topdir + "/tmp-sstatesamehash/stamps/" + k + "." + files1[k]
+ sigdatafile2 = self.topdir + "/tmp-sstatesamehash2/stamps/" + k + "." + files2[k]
+ output = bb.siggen.compare_sigfiles(sigdatafile1, sigdatafile2)
+ if output:
+ print('\n'.join(output))
elif k in files1 and k not in files2:
print("%s in files1" % k)
elif k not in files1 and k in files2:
--
2.12.3
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ
2017-08-04 18:57 [PATCH 1/2] sstatetests: Use python function instead of bitbake-diffsigs script leonardo.sandoval.gonzalez
@ 2017-08-04 18:57 ` leonardo.sandoval.gonzalez
2017-08-07 16:32 ` Burton, Ross
0 siblings, 1 reply; 6+ messages in thread
From: leonardo.sandoval.gonzalez @ 2017-08-04 18:57 UTC (permalink / raw)
To: openembedded-core
From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
For perfomance reasons, limit the number of signature comparisons when
stamps differ. The limit set is hardcoded to 20.
[YOCTO #11651]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
meta/lib/oeqa/selftest/cases/sstatetests.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
index 0b36027918..695e2a86d5 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -468,9 +468,15 @@ http_proxy = "http://example.com/"
# No changes, so we're done
return
+ i_sigfile, max_sigfiles_to_compare = 0, 20
for k in files1.keys() | files2.keys():
if k in files1 and k in files2:
+ i_sigfile += 1
print("%s differs:" % k)
+ # this is an expensive computation, thus just compare the first
+ # 'max_sigfiles_to_compare' k files
+ if i_sigfile > max_sigfiles_to_compare:
+ continue
sigdatafile1 = self.topdir + "/tmp-sstatesamehash/stamps/" + k + "." + files1[k]
sigdatafile2 = self.topdir + "/tmp-sstatesamehash2/stamps/" + k + "." + files2[k]
output = bb.siggen.compare_sigfiles(sigdatafile1, sigdatafile2)
--
2.12.3
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ
2017-08-04 18:57 ` [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ leonardo.sandoval.gonzalez
@ 2017-08-07 16:32 ` Burton, Ross
2017-08-07 17:06 ` Leonardo Sandoval
0 siblings, 1 reply; 6+ messages in thread
From: Burton, Ross @ 2017-08-07 16:32 UTC (permalink / raw)
To: Leonardo Sandoval; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]
On 4 August 2017 at 19:57, <leonardo.sandoval.gonzalez@linux.intel.com>
wrote:
> + i_sigfile, max_sigfiles_to_compare = 0, 20
> for k in files1.keys() | files2.keys():
> if k in files1 and k in files2:
> + i_sigfile += 1
> print("%s differs:" % k)
> + # this is an expensive computation, thus just compare the
> first
> + # 'max_sigfiles_to_compare' k files
> + if i_sigfile > max_sigfiles_to_compare:
> + continue
> sigdatafile1 = self.topdir +
> "/tmp-sstatesamehash/stamps/" + k + "." + files1[k]
> sigdatafile2 = self.topdir +
> "/tmp-sstatesamehash2/stamps/" + k + "." + files2[k]
> output = bb.siggen.compare_sigfiles(sigdatafile1,
> sigdatafile2)
>
A neater way to do this:
-for k in files1.keys() | files2.keys():
+for k in list(files1.keys() | files2.key())[:20];
(take the union of the files1 and files2 keys, turn it into a list, and
slice off up to the first 20)
Ross
[-- Attachment #2: Type: text/html, Size: 1793 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ
2017-08-07 17:06 ` Leonardo Sandoval
@ 2017-08-07 17:05 ` Burton, Ross
2017-08-07 17:19 ` Leonardo Sandoval
0 siblings, 1 reply; 6+ messages in thread
From: Burton, Ross @ 2017-08-07 17:05 UTC (permalink / raw)
To: Leonardo Sandoval; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 534 bytes --]
On 7 August 2017 at 18:06, Leonardo Sandoval <
leonardo.sandoval.gonzalez@linux.intel.com> wrote:
> yes, but we this way, we are not listing those > 20 and I believe this
> is important. This fix just diffsigs the first 20, and the rest are just
> listed for convenience.
>
Good point. You could neatly print the filenames in one iteration, and
then the first 20 diffs in another loop.
also, sets are not 'subscriptable' thus slicing is not possible on sets.
>
Which is why I called list() before slicing.
Ross
[-- Attachment #2: Type: text/html, Size: 1050 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ
2017-08-07 16:32 ` Burton, Ross
@ 2017-08-07 17:06 ` Leonardo Sandoval
2017-08-07 17:05 ` Burton, Ross
0 siblings, 1 reply; 6+ messages in thread
From: Leonardo Sandoval @ 2017-08-07 17:06 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE-core
On Mon, 2017-08-07 at 17:32 +0100, Burton, Ross wrote:
> On 4 August 2017 at 19:57,
> <leonardo.sandoval.gonzalez@linux.intel.com> wrote:
> + i_sigfile, max_sigfiles_to_compare = 0, 20
> for k in files1.keys() | files2.keys():
> if k in files1 and k in files2:
> + i_sigfile += 1
> print("%s differs:" % k)
> + # this is an expensive computation, thus just
> compare the first
> + # 'max_sigfiles_to_compare' k files
> + if i_sigfile > max_sigfiles_to_compare:
> + continue
> sigdatafile1 = self.topdir +
> "/tmp-sstatesamehash/stamps/" + k + "." + files1[k]
> sigdatafile2 = self.topdir +
> "/tmp-sstatesamehash2/stamps/" + k + "." + files2[k]
> output =
> bb.siggen.compare_sigfiles(sigdatafile1, sigdatafile2)
>
>
> A neater way to do this:
>
>
> -for k in files1.keys() | files2.keys():
> +for k in list(files1.keys() | files2.key())[:20];
>
>
yes, but we this way, we are not listing those > 20 and I believe this
is important. This fix just diffsigs the first 20, and the rest are just
listed for convenience.
also, sets are not 'subscriptable' thus slicing is not possible on sets.
Leo
> (take the union of the files1 and files2 keys, turn it into a list,
> and slice off up to the first 20)
>
>
> Ross
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ
2017-08-07 17:05 ` Burton, Ross
@ 2017-08-07 17:19 ` Leonardo Sandoval
0 siblings, 0 replies; 6+ messages in thread
From: Leonardo Sandoval @ 2017-08-07 17:19 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE-core
On Mon, 2017-08-07 at 18:05 +0100, Burton, Ross wrote:
> On 7 August 2017 at 18:06, Leonardo Sandoval
> <leonardo.sandoval.gonzalez@linux.intel.com> wrote:
> yes, but we this way, we are not listing those > 20 and I
> believe this
> is important. This fix just diffsigs the first 20, and the
> rest are just
> listed for convenience.
>
>
> Good point. You could neatly print the filenames in one iteration,
> and then the first 20 diffs in another loop.
not super convinced but yes, that can be a another way to avoid the if
inside the for body. I will send a v2.
Leo
>
>
> also, sets are not 'subscriptable' thus slicing is not
> possible on sets.
>
>
> Which is why I called list() before slicing.
>
>
> Ross
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-08-07 17:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-08-04 18:57 [PATCH 1/2] sstatetests: Use python function instead of bitbake-diffsigs script leonardo.sandoval.gonzalez
2017-08-04 18:57 ` [PATCH 2/2] sstatetests: limit the number of signature comparisons when differ leonardo.sandoval.gonzalez
2017-08-07 16:32 ` Burton, Ross
2017-08-07 17:06 ` Leonardo Sandoval
2017-08-07 17:05 ` Burton, Ross
2017-08-07 17:19 ` Leonardo Sandoval
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox