* QA cycle report for 2.2.2 rc2
@ 2017-09-14 23:53 Cruz, Libertad
2017-09-15 16:50 ` Richard Purdie
0 siblings, 1 reply; 8+ messages in thread
From: Cruz, Libertad @ 2017-09-14 23:53 UTC (permalink / raw)
To: yocto@yoctoproject.org, Jolley, Stephen K, Zhang, Jessica,
Purdie, Richard
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
[-- Attachment #1: Type: text/plain, Size: 3473 bytes --]
Hello All,
Here is the report for the 2.2.2 point release test cycle.
Full Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-_Full_Point_Release_Test_Cycle_2.2.2_rc2
======= Summary ========
The QA cycle for point release 2.2.2 rc2 is complete. There are 3 new bugs: bug 12080[2] blocks testing to the eSDK component since eSDK functionalities cannot be executed. Bugs 12075[1], 12073[3], 12081[4] don't block full functionalities.
LPT results are waiting to be updated by Windriver.
Performance
Compared with 2.2.2 rc1 there was a regression of 14.71 while building rootfs on Ubuntu, there was also a regression on Fedora while building sato of 7.46% and a regression on kernel of 12.12%.
Ubuntu Test 2.2.2_rc1 2.2.2_rc2 %
sato 1:00:42 1:01:16 0.93
rootfs 2:16 2:36 14.71
rmwork 0:59:29 0:59:51 0.62
kernel 5:01 5:00 -0.33
eSDK 3:22 3:19 -1.49
Fedora Test 2.2.2_rc1 2.2.2_rc2 %
sato 1:01:55 1:06:32 7.46
rootfs 2:49 2:51 1.18
rmwork 0:59:52 1:01:32 2.78
kernel 6:03 6:47 12.12
eSDK 3:44 3:54 4.46
ptest
There was an improvement on the pass rate in the following packages in comparison with 2.2.2 rc1:
gdk-pixbuf with 5%
libxml2 with 4.20%
lttng-tools with 3.20%
On the other side there was a regression on the pass rate in comparison with 2.2.2 rc1:
acl with 6.80%
e2fsprogs with 100% regression
gawk with .10%
quilt with 1.80%
valgrind of 53.10%
Compliance
Results are yet to be documented by Windriver.
======= QA-Hints========
There is only one concern on the eSDK component it cannot execute. QA recommends an 2.2.2 rc3 to only verify the eSDK component.
======= Bugs ========
New Bugs
-12075[1] [morty] testimage quemux86/qemux86-64 lsb tcgetattr: Input/output error^M
-12080[2] [Morty][Test Case 1605] TCTEMP_2.3_AUTO_sdkext_eSDK_devtool_build_make
-12073[3] selftest: setUpClass failed failing on ubuntu 17.04 (oeqa.selftest.signing.Signing)
-12081[4] [morty] PTEST: Package Name flex, gzip, slang, do not appear in 2.2.2 rc2
Full Bug Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-_Full_Point_Release_Test_Cycle_2.2.2_rc2#Bugs_Found_during_QA_Test
======== Links =========
1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12075 [1]
2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12080 [2]
3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12073 [3]
4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12081 [4]
Regards
Libertad G.
[-- Attachment #2: Type: text/html, Size: 19752 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-14 23:53 QA cycle report for 2.2.2 rc2 Cruz, Libertad
@ 2017-09-15 16:50 ` Richard Purdie
2017-09-15 18:04 ` Bystricky, Juro
0 siblings, 1 reply; 8+ messages in thread
From: Richard Purdie @ 2017-09-15 16:50 UTC (permalink / raw)
To: Cruz, Libertad, yocto@yoctoproject.org, Jolley, Stephen K,
Zhang, Jessica, juro.bystricky
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
This is a tricky one to evaluate. Going through the bugs in the QA
report:
Bug 12075 - think this is ok to fix on the branch and release note
Bug 12080 - we think minimal eSDK never worked in morty therefore this
is a QA testing error?
Bug 12073 - we continue to see various gpg errors even with master,
propose not to block release on this and only fix master
Bug 12009 - resolved and already fixed in 2.2.2 (I marked it resolved)
Bug 11150 - was a bug with 2.2.1 but should be fixed in 2.2.2?
Bug 11611 - was this found in 2.2.2?
Bug 11064 - marked as resolved as per QA comment, fix was in 2.2.2
Bug 11597 - fix was added to 2.2.2 (marked as resolved)
Bug 10460 - open and unresolved with master but not release blocker IMO
Bug 11797 - can't see any backport but not a release blocker as the
source mirrors work
Bug 11109 - No backport afaict but not release blocker
Which leave us with 12081 to understand before we can decide whether to
release this or not.
So the decision to release or not depends on QA agreeing with the resolution of 12080 and figuring out what happened with 12081.
[Thanks to Ross for helping with some of this]
Cheers,
Richard
On Thu, 2017-09-14 at 16:53 -0700, Cruz, Libertad wrote:
> Here is the report for the 2.2.2 point release test cycle.
> Full Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-
> _Full_Point_Release_Test_Cycle_2.2.2_rc2
> ======= Summary ========
> The QA cycle for point release 2.2.2 rc2 is complete. There are 3
> new bugs: bug 12080[2] blocks testing to the eSDK component since
> eSDK functionalities cannot be executed. Bugs 12075[1], 12073[3],
> 12081[4] don’t block full functionalities.
>
> LPT results are waiting to be updated by Windriver.
>
> Performance
> Compared with 2.2.2 rc1 there was a regression of 14.71 while
> building rootfs on Ubuntu, there was also a regression on Fedora
> while building sato of 7.46% and a regression on kernel of 12.12%.
>
>
> Ubuntu Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:00:42
> 1:01:16 0.93
> rootfs 2:16
> 2:36 14.71
> rmwork 0:59:29 0:59:51
> 0.62
> kernel 5:01
> 5:00 -0.33
> eSDK 3:22
> 3:19 -1.49
>
>
>
>
> Fedora Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:01:55
> 1:06:32 7.46
> rootfs 2:49
> 2:51 1.18
> rmwork 0:59:52 1:01:32
> 2.78
> kernel 6:03
> 6:47 12.12
> eSDK 3:44
> 3:54 4.46
>
>
> ptest
> There was an improvement on the pass rate in the following packages
> in comparison with 2.2.2 rc1:
> gdk-pixbuf with 5%
> libxml2 with 4.20%
> lttng-tools with 3.20%
>
> On the other side there was a regression on the pass rate in
> comparison with 2.2.2 rc1:
> acl with 6.80%
> e2fsprogs with 100% regression
> gawk with .10%
> quilt with 1.80%
> valgrind of 53.10%
>
>
> Compliance
> Results are yet to be documented by Windriver.
>
> ======= QA-Hints========
>
> There is only one concern on the eSDK component it cannot execute. QA
> recommends an 2.2.2 rc3 to only verify the eSDK component.
>
> ======= Bugs ========
>
> New Bugs
> -12075[1] [morty] testimage quemux86/qemux86-64 lsb
> tcgetattr: Input/output error^M
> -12080[2] [Morty][Test Case 1605]
> TCTEMP_2.3_AUTO_sdkext_eSDK_devtool_build_make
> -12073[3] selftest: setUpClass failed failing on ubuntu
> 17.04 (oeqa.selftest.signing.Signing)
> -12081[4] [morty] PTEST: Package Name flex, gzip, slang,
> do not appear in 2.2.2 rc2
>
> Full Bug Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-1
> 3_-_Full_Point_Release_Test_Cycle_2.2.2_rc2#Bugs_Found_during_QA_Test
>
>
> ======== Links =========
>
> 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12075 [1]
> 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12080 [2]
> 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12073 [3]
> 4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12081 [4]
>
>
>
> Regards
> Libertad G.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-15 16:50 ` Richard Purdie
@ 2017-09-15 18:04 ` Bystricky, Juro
2017-09-15 19:50 ` Cruz, Libertad
0 siblings, 1 reply; 8+ messages in thread
From: Bystricky, Juro @ 2017-09-15 18:04 UTC (permalink / raw)
To: Richard Purdie, Cruz, Libertad, yocto@yoctoproject.org,
Jolley, Stephen K, Zhang, Jessica
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
Regarding 12081:
Looking at some previous ptest results in:
https://wiki.yoctoproject.org/wiki/QA_sanity_history
we can see that the offending tests (gzip, flex, slang) were not present in:
https://wiki.yoctoproject.org/wiki/Ptest_62d7d4130202d8ede16abf9e7d779361ca70847e
but started to appear after
https://wiki.yoctoproject.org/wiki/Ptest_6e20b31d5d17133e0fca086e12a0ad06ab5c4cc8 (flex)
Also checking the manifest file, the offending test were also missing in
http://downloads.yoctoproject.org/releases/yocto/yocto-2.2.1/machines/genericx86-64/core-image-sato-sdk-ptest-genericx86-64.manifest
So this is not a regression issue.
Juro
From: Richard Purdie [richard.purdie@linuxfoundation.org]
Sent: Friday, September 15, 2017 9:50 AM
To: Cruz, Libertad; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica; Bystricky, Juro
Cc: Perez Carranza, Jose; Cruz Alcaraz, Juan M; Jordan, Robin L
Subject: Re: QA cycle report for 2.2.2 rc2
This is a tricky one to evaluate. Going through the bugs in the QA
report:
Bug 12075 - think this is ok to fix on the branch and release note
Bug 12080 - we think minimal eSDK never worked in morty therefore this
is a QA testing error?
Bug 12073 - we continue to see various gpg errors even with master,
propose not to block release on this and only fix master
Bug 12009 - resolved and already fixed in 2.2.2 (I marked it resolved)
Bug 11150 - was a bug with 2.2.1 but should be fixed in 2.2.2?
Bug 11611 - was this found in 2.2.2?
Bug 11064 - marked as resolved as per QA comment, fix was in 2.2.2
Bug 11597 - fix was added to 2.2.2 (marked as resolved)
Bug 10460 - open and unresolved with master but not release blocker IMO
Bug 11797 - can't see any backport but not a release blocker as the
source mirrors work
Bug 11109 - No backport afaict but not release blocker
Which leave us with 12081 to understand before we can decide whether to
release this or not.
So the decision to release or not depends on QA agreeing with the resolution of 12080 and figuring out what happened with 12081.
[Thanks to Ross for helping with some of this]
Cheers,
Richard
On Thu, 2017-09-14 at 16:53 -0700, Cruz, Libertad wrote:
> Here is the report for the 2.2.2 point release test cycle.
> Full Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-
> _Full_Point_Release_Test_Cycle_2.2.2_rc2
> ======= Summary ========
> The QA cycle for point release 2.2.2 rc2 is complete. There are 3
> new bugs: bug 12080[2] blocks testing to the eSDK component since
> eSDK functionalities cannot be executed. Bugs 12075[1], 12073[3],
> 12081[4] don’t block full functionalities.
>
> LPT results are waiting to be updated by Windriver.
>
> Performance
> Compared with 2.2.2 rc1 there was a regression of 14.71 while
> building rootfs on Ubuntu, there was also a regression on Fedora
> while building sato of 7.46% and a regression on kernel of 12.12%.
>
>
> Ubuntu Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:00:42
> 1:01:16 0.93
> rootfs 2:16
> 2:36 14.71
> rmwork 0:59:29 0:59:51
> 0.62
> kernel 5:01
> 5:00 -0.33
> eSDK 3:22
> 3:19 -1.49
>
>
>
>
> Fedora Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:01:55
> 1:06:32 7.46
> rootfs 2:49
> 2:51 1.18
> rmwork 0:59:52 1:01:32
> 2.78
> kernel 6:03
> 6:47 12.12
> eSDK 3:44
> 3:54 4.46
>
>
> ptest
> There was an improvement on the pass rate in the following packages
> in comparison with 2.2.2 rc1:
> gdk-pixbuf with 5%
> libxml2 with 4.20%
> lttng-tools with 3.20%
>
> On the other side there was a regression on the pass rate in
> comparison with 2.2.2 rc1:
> acl with 6.80%
> e2fsprogs with 100% regression
> gawk with .10%
> quilt with 1.80%
> valgrind of 53.10%
>
>
> Compliance
> Results are yet to be documented by Windriver.
>
> ======= QA-Hints========
>
> There is only one concern on the eSDK component it cannot execute. QA
> recommends an 2.2.2 rc3 to only verify the eSDK component.
>
> ======= Bugs ========
>
> New Bugs
> -12075[1] [morty] testimage quemux86/qemux86-64 lsb
> tcgetattr: Input/output error^M
> -12080[2] [Morty][Test Case 1605]
> TCTEMP_2.3_AUTO_sdkext_eSDK_devtool_build_make
> -12073[3] selftest: setUpClass failed failing on ubuntu
> 17.04 (oeqa.selftest.signing.Signing)
> -12081[4] [morty] PTEST: Package Name flex, gzip, slang,
> do not appear in 2.2.2 rc2
>
> Full Bug Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-1
> 3_-_Full_Point_Release_Test_Cycle_2.2.2_rc2#Bugs_Found_during_QA_Test
>
>
> ======== Links =========
>
> 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12075 [1]
> 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12080 [2]
> 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12073 [3]
> 4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12081 [4]
>
>
>
> Regards
> Libertad G.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-15 18:04 ` Bystricky, Juro
@ 2017-09-15 19:50 ` Cruz, Libertad
2017-09-15 20:01 ` Bystricky, Juro
0 siblings, 1 reply; 8+ messages in thread
From: Cruz, Libertad @ 2017-09-15 19:50 UTC (permalink / raw)
To: Bystricky, Juro, Richard Purdie, yocto@yoctoproject.org,
Jolley, Stephen K, Zhang, Jessica
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
Hello,
Regarding 12080: this was a QA documentation error on the test steps, verified it again and changed the bug to not a bug, documented the correct steps for the QA.
Regarding 12081: To open bugs on ptest we make a comparison from the last run in this case the comparison was made with 2.2.2 rc1 Ptest 5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 which in turn was compared with Ptest 9ed748a542b520c1cb763d981969233c0f5efd4e which was compared with Ptest 8e15e9b6e478f6368034519b2a8fd3c7ea71d23b which was compared with Ptest 6bd890d9e011014cf323e61267f8b256949d44aa, they all hold this packages.
Best Regards
Libertad G.
-----Original Message-----
From: Bystricky, Juro
Sent: Friday, September 15, 2017 1:04 PM
To: Richard Purdie <richard.purdie@linuxfoundation.org>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Cruz Alcaraz, Juan M <juan.m.cruz.alcaraz@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>
Subject: RE: QA cycle report for 2.2.2 rc2
Regarding 12081:
Looking at some previous ptest results in:
https://wiki.yoctoproject.org/wiki/QA_sanity_history
we can see that the offending tests (gzip, flex, slang) were not present in:
https://wiki.yoctoproject.org/wiki/Ptest_62d7d4130202d8ede16abf9e7d779361ca70847e
but started to appear after
https://wiki.yoctoproject.org/wiki/Ptest_6e20b31d5d17133e0fca086e12a0ad06ab5c4cc8 (flex)
Also checking the manifest file, the offending test were also missing in http://downloads.yoctoproject.org/releases/yocto/yocto-2.2.1/machines/genericx86-64/core-image-sato-sdk-ptest-genericx86-64.manifest
So this is not a regression issue.
Juro
From: Richard Purdie [richard.purdie@linuxfoundation.org]
Sent: Friday, September 15, 2017 9:50 AM
To: Cruz, Libertad; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica; Bystricky, Juro
Cc: Perez Carranza, Jose; Cruz Alcaraz, Juan M; Jordan, Robin L
Subject: Re: QA cycle report for 2.2.2 rc2
This is a tricky one to evaluate. Going through the bugs in the QA
report:
Bug 12075 - think this is ok to fix on the branch and release note
Bug 12080 - we think minimal eSDK never worked in morty therefore this
is a QA testing error?
Bug 12073 - we continue to see various gpg errors even with master,
propose not to block release on this and only fix master
Bug 12009 - resolved and already fixed in 2.2.2 (I marked it resolved)
Bug 11150 - was a bug with 2.2.1 but should be fixed in 2.2.2?
Bug 11611 - was this found in 2.2.2?
Bug 11064 - marked as resolved as per QA comment, fix was in 2.2.2
Bug 11597 - fix was added to 2.2.2 (marked as resolved)
Bug 10460 - open and unresolved with master but not release blocker IMO
Bug 11797 - can't see any backport but not a release blocker as the
source mirrors work
Bug 11109 - No backport afaict but not release blocker
Which leave us with 12081 to understand before we can decide whether to release this or not.
So the decision to release or not depends on QA agreeing with the resolution of 12080 and figuring out what happened with 12081.
[Thanks to Ross for helping with some of this]
Cheers,
Richard
On Thu, 2017-09-14 at 16:53 -0700, Cruz, Libertad wrote:
> Here is the report for the 2.2.2 point release test cycle.
> Full Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-
> _Full_Point_Release_Test_Cycle_2.2.2_rc2
> ======= Summary ========
> The QA cycle for point release 2.2.2 rc2 is complete. There are 3 new
> bugs: bug 12080[2] blocks testing to the eSDK component since eSDK
> functionalities cannot be executed. Bugs 12075[1], 12073[3], 12081[4]
> don't block full functionalities.
>
> LPT results are waiting to be updated by Windriver.
>
> Performance
> Compared with 2.2.2 rc1 there was a regression of 14.71 while building
> rootfs on Ubuntu, there was also a regression on Fedora while building
> sato of 7.46% and a regression on kernel of 12.12%.
>
>
> Ubuntu Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:00:42
> 1:01:16 0.93
> rootfs 2:16
> 2:36 14.71
> rmwork 0:59:29 0:59:51
> 0.62
> kernel 5:01
> 5:00 -0.33
> eSDK 3:22
> 3:19 -1.49
>
>
>
>
> Fedora Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:01:55
> 1:06:32 7.46
> rootfs 2:49
> 2:51 1.18
> rmwork 0:59:52 1:01:32
> 2.78
> kernel 6:03
> 6:47 12.12
> eSDK 3:44
> 3:54 4.46
>
>
> ptest
> There was an improvement on the pass rate in the following packages in
> comparison with 2.2.2 rc1:
> gdk-pixbuf with 5%
> libxml2 with 4.20%
> lttng-tools with 3.20%
>
> On the other side there was a regression on the pass rate in
> comparison with 2.2.2 rc1:
> acl with 6.80%
> e2fsprogs with 100% regression
> gawk with .10%
> quilt with 1.80%
> valgrind of 53.10%
>
>
> Compliance
> Results are yet to be documented by Windriver.
>
> ======= QA-Hints========
>
> There is only one concern on the eSDK component it cannot execute. QA
> recommends an 2.2.2 rc3 to only verify the eSDK component.
>
> ======= Bugs ========
>
> New Bugs
> -12075[1] [morty] testimage quemux86/qemux86-64 lsb
> tcgetattr: Input/output error^M
> -12080[2] [Morty][Test Case 1605]
> TCTEMP_2.3_AUTO_sdkext_eSDK_devtool_build_make
> -12073[3] selftest: setUpClass failed failing on ubuntu
> 17.04 (oeqa.selftest.signing.Signing)
> -12081[4] [morty] PTEST: Package Name flex, gzip, slang,
> do not appear in 2.2.2 rc2
>
> Full Bug Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-1
> 3_-_Full_Point_Release_Test_Cycle_2.2.2_rc2#Bugs_Found_during_QA_Test
>
>
> ======== Links =========
>
> 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12075 [1]
> 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12080 [2]
> 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12073 [3]
> 4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12081 [4]
>
>
>
> Regards
> Libertad G.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-15 19:50 ` Cruz, Libertad
@ 2017-09-15 20:01 ` Bystricky, Juro
2017-09-15 20:06 ` akuster808
0 siblings, 1 reply; 8+ messages in thread
From: Bystricky, Juro @ 2017-09-15 20:01 UTC (permalink / raw)
To: Cruz, Libertad, Richard Purdie, yocto@yoctoproject.org,
Jolley, Stephen K, Zhang, Jessica
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
The commit is for 2.4 M3 (5f6945f5031e1a4ca116cc1eccf4c2f9dc228547)
2.4 does contain the ptests for gzip, slang and flex. (Although there are some other mysteries, like sed-ptest is missing)
It would seem that 2.2 had never contained (or lost a while ago) those ptests.
Juro
________________________________________
From: Cruz, Libertad
Sent: Friday, September 15, 2017 12:50 PM
To: Bystricky, Juro; Richard Purdie; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica
Cc: Perez Carranza, Jose; Cruz Alcaraz, Juan M; Jordan, Robin L
Subject: RE: QA cycle report for 2.2.2 rc2
Hello,
Regarding 12080: this was a QA documentation error on the test steps, verified it again and changed the bug to not a bug, documented the correct steps for the QA.
Regarding 12081: To open bugs on ptest we make a comparison from the last run in this case the comparison was made with 2.2.2 rc1 Ptest 5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 which in turn was compared with Ptest 9ed748a542b520c1cb763d981969233c0f5efd4e which was compared with Ptest 8e15e9b6e478f6368034519b2a8fd3c7ea71d23b which was compared with Ptest 6bd890d9e011014cf323e61267f8b256949d44aa, they all hold this packages.
Best Regards
Libertad G.
-----Original Message-----
From: Bystricky, Juro
Sent: Friday, September 15, 2017 1:04 PM
To: Richard Purdie <richard.purdie@linuxfoundation.org>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Cruz Alcaraz, Juan M <juan.m.cruz.alcaraz@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>
Subject: RE: QA cycle report for 2.2.2 rc2
Regarding 12081:
Looking at some previous ptest results in:
https://wiki.yoctoproject.org/wiki/QA_sanity_history
we can see that the offending tests (gzip, flex, slang) were not present in:
https://wiki.yoctoproject.org/wiki/Ptest_62d7d4130202d8ede16abf9e7d779361ca70847e
but started to appear after
https://wiki.yoctoproject.org/wiki/Ptest_6e20b31d5d17133e0fca086e12a0ad06ab5c4cc8 (flex)
Also checking the manifest file, the offending test were also missing in http://downloads.yoctoproject.org/releases/yocto/yocto-2.2.1/machines/genericx86-64/core-image-sato-sdk-ptest-genericx86-64.manifest
So this is not a regression issue.
Juro
From: Richard Purdie [richard.purdie@linuxfoundation.org]
Sent: Friday, September 15, 2017 9:50 AM
To: Cruz, Libertad; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica; Bystricky, Juro
Cc: Perez Carranza, Jose; Cruz Alcaraz, Juan M; Jordan, Robin L
Subject: Re: QA cycle report for 2.2.2 rc2
This is a tricky one to evaluate. Going through the bugs in the QA
report:
Bug 12075 - think this is ok to fix on the branch and release note
Bug 12080 - we think minimal eSDK never worked in morty therefore this
is a QA testing error?
Bug 12073 - we continue to see various gpg errors even with master,
propose not to block release on this and only fix master
Bug 12009 - resolved and already fixed in 2.2.2 (I marked it resolved)
Bug 11150 - was a bug with 2.2.1 but should be fixed in 2.2.2?
Bug 11611 - was this found in 2.2.2?
Bug 11064 - marked as resolved as per QA comment, fix was in 2.2.2
Bug 11597 - fix was added to 2.2.2 (marked as resolved)
Bug 10460 - open and unresolved with master but not release blocker IMO
Bug 11797 - can't see any backport but not a release blocker as the
source mirrors work
Bug 11109 - No backport afaict but not release blocker
Which leave us with 12081 to understand before we can decide whether to release this or not.
So the decision to release or not depends on QA agreeing with the resolution of 12080 and figuring out what happened with 12081.
[Thanks to Ross for helping with some of this]
Cheers,
Richard
On Thu, 2017-09-14 at 16:53 -0700, Cruz, Libertad wrote:
> Here is the report for the 2.2.2 point release test cycle.
> Full Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-
> _Full_Point_Release_Test_Cycle_2.2.2_rc2
> ======= Summary ========
> The QA cycle for point release 2.2.2 rc2 is complete. There are 3 new
> bugs: bug 12080[2] blocks testing to the eSDK component since eSDK
> functionalities cannot be executed. Bugs 12075[1], 12073[3], 12081[4]
> don’t block full functionalities.
>
> LPT results are waiting to be updated by Windriver.
>
> Performance
> Compared with 2.2.2 rc1 there was a regression of 14.71 while building
> rootfs on Ubuntu, there was also a regression on Fedora while building
> sato of 7.46% and a regression on kernel of 12.12%.
>
>
> Ubuntu Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:00:42
> 1:01:16 0.93
> rootfs 2:16
> 2:36 14.71
> rmwork 0:59:29 0:59:51
> 0.62
> kernel 5:01
> 5:00 -0.33
> eSDK 3:22
> 3:19 -1.49
>
>
>
>
> Fedora Test 2.2.2_rc1
> 2.2.2_rc2 %
> sato 1:01:55
> 1:06:32 7.46
> rootfs 2:49
> 2:51 1.18
> rmwork 0:59:52 1:01:32
> 2.78
> kernel 6:03
> 6:47 12.12
> eSDK 3:44
> 3:54 4.46
>
>
> ptest
> There was an improvement on the pass rate in the following packages in
> comparison with 2.2.2 rc1:
> gdk-pixbuf with 5%
> libxml2 with 4.20%
> lttng-tools with 3.20%
>
> On the other side there was a regression on the pass rate in
> comparison with 2.2.2 rc1:
> acl with 6.80%
> e2fsprogs with 100% regression
> gawk with .10%
> quilt with 1.80%
> valgrind of 53.10%
>
>
> Compliance
> Results are yet to be documented by Windriver.
>
> ======= QA-Hints========
>
> There is only one concern on the eSDK component it cannot execute. QA
> recommends an 2.2.2 rc3 to only verify the eSDK component.
>
> ======= Bugs ========
>
> New Bugs
> -12075[1] [morty] testimage quemux86/qemux86-64 lsb
> tcgetattr: Input/output error^M
> -12080[2] [Morty][Test Case 1605]
> TCTEMP_2.3_AUTO_sdkext_eSDK_devtool_build_make
> -12073[3] selftest: setUpClass failed failing on ubuntu
> 17.04 (oeqa.selftest.signing.Signing)
> -12081[4] [morty] PTEST: Package Name flex, gzip, slang,
> do not appear in 2.2.2 rc2
>
> Full Bug Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-1
> 3_-_Full_Point_Release_Test_Cycle_2.2.2_rc2#Bugs_Found_during_QA_Test
>
>
> ======== Links =========
>
> 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12075 [1]
> 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12080 [2]
> 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12073 [3]
> 4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12081 [4]
>
>
>
> Regards
> Libertad G.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-15 20:01 ` Bystricky, Juro
@ 2017-09-15 20:06 ` akuster808
2017-09-15 20:14 ` Bystricky, Juro
0 siblings, 1 reply; 8+ messages in thread
From: akuster808 @ 2017-09-15 20:06 UTC (permalink / raw)
To: Bystricky, Juro, Cruz, Libertad, Richard Purdie,
yocto@yoctoproject.org, Jolley, Stephen K, Zhang, Jessica
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
On 09/15/2017 01:01 PM, Bystricky, Juro wrote:
> The commit is for 2.4 M3 (5f6945f5031e1a4ca116cc1eccf4c2f9dc228547)
> 2.4 does contain the ptests for gzip, slang and flex. (Although there are some other mysteries, like sed-ptest is missing)
> It would seem that 2.2 had never contained (or lost a while ago) those ptests.
Thanks for looking into this. Is there any action I need to follow up on?
- armin
>
> Juro
> ________________________________________
> From: Cruz, Libertad
> Sent: Friday, September 15, 2017 12:50 PM
> To: Bystricky, Juro; Richard Purdie; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica
> Cc: Perez Carranza, Jose; Cruz Alcaraz, Juan M; Jordan, Robin L
> Subject: RE: QA cycle report for 2.2.2 rc2
>
> Hello,
>
>
> Regarding 12080: this was a QA documentation error on the test steps, verified it again and changed the bug to not a bug, documented the correct steps for the QA.
>
> Regarding 12081: To open bugs on ptest we make a comparison from the last run in this case the comparison was made with 2.2.2 rc1 Ptest 5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 which in turn was compared with Ptest 9ed748a542b520c1cb763d981969233c0f5efd4e which was compared with Ptest 8e15e9b6e478f6368034519b2a8fd3c7ea71d23b which was compared with Ptest 6bd890d9e011014cf323e61267f8b256949d44aa, they all hold this packages.
>
>
>
>
> Best Regards
> Libertad G.
>
> -----Original Message-----
> From: Bystricky, Juro
> Sent: Friday, September 15, 2017 1:04 PM
> To: Richard Purdie <richard.purdie@linuxfoundation.org>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
> Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Cruz Alcaraz, Juan M <juan.m.cruz.alcaraz@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>
> Subject: RE: QA cycle report for 2.2.2 rc2
>
>
> Regarding 12081:
> Looking at some previous ptest results in:
> https://wiki.yoctoproject.org/wiki/QA_sanity_history
> we can see that the offending tests (gzip, flex, slang) were not present in:
> https://wiki.yoctoproject.org/wiki/Ptest_62d7d4130202d8ede16abf9e7d779361ca70847e
> but started to appear after
> https://wiki.yoctoproject.org/wiki/Ptest_6e20b31d5d17133e0fca086e12a0ad06ab5c4cc8 (flex)
>
> Also checking the manifest file, the offending test were also missing in http://downloads.yoctoproject.org/releases/yocto/yocto-2.2.1/machines/genericx86-64/core-image-sato-sdk-ptest-genericx86-64.manifest
>
> So this is not a regression issue.
>
> Juro
>
> From: Richard Purdie [richard.purdie@linuxfoundation.org]
> Sent: Friday, September 15, 2017 9:50 AM
> To: Cruz, Libertad; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica; Bystricky, Juro
> Cc: Perez Carranza, Jose; Cruz Alcaraz, Juan M; Jordan, Robin L
> Subject: Re: QA cycle report for 2.2.2 rc2
>
> This is a tricky one to evaluate. Going through the bugs in the QA
> report:
>
> Bug 12075 - think this is ok to fix on the branch and release note
> Bug 12080 - we think minimal eSDK never worked in morty therefore this
> is a QA testing error?
> Bug 12073 - we continue to see various gpg errors even with master,
> propose not to block release on this and only fix master
> Bug 12009 - resolved and already fixed in 2.2.2 (I marked it resolved)
> Bug 11150 - was a bug with 2.2.1 but should be fixed in 2.2.2?
> Bug 11611 - was this found in 2.2.2?
> Bug 11064 - marked as resolved as per QA comment, fix was in 2.2.2
> Bug 11597 - fix was added to 2.2.2 (marked as resolved)
> Bug 10460 - open and unresolved with master but not release blocker IMO
> Bug 11797 - can't see any backport but not a release blocker as the
> source mirrors work
> Bug 11109 - No backport afaict but not release blocker
>
> Which leave us with 12081 to understand before we can decide whether to release this or not.
>
> So the decision to release or not depends on QA agreeing with the resolution of 12080 and figuring out what happened with 12081.
>
> [Thanks to Ross for helping with some of this]
>
> Cheers,
>
> Richard
>
>
>
>
> On Thu, 2017-09-14 at 16:53 -0700, Cruz, Libertad wrote:
>> Here is the report for the 2.2.2 point release test cycle.
>> Full Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-13_-
>> _Full_Point_Release_Test_Cycle_2.2.2_rc2
>> ======= Summary ========
>> The QA cycle for point release 2.2.2 rc2 is complete. There are 3 new
>> bugs: bug 12080[2] blocks testing to the eSDK component since eSDK
>> functionalities cannot be executed. Bugs 12075[1], 12073[3], 12081[4]
>> don’t block full functionalities.
>>
>> LPT results are waiting to be updated by Windriver.
>>
>> Performance
>> Compared with 2.2.2 rc1 there was a regression of 14.71 while building
>> rootfs on Ubuntu, there was also a regression on Fedora while building
>> sato of 7.46% and a regression on kernel of 12.12%.
>>
>>
>> Ubuntu Test 2.2.2_rc1
>> 2.2.2_rc2 %
>> sato 1:00:42
>> 1:01:16 0.93
>> rootfs 2:16
>> 2:36 14.71
>> rmwork 0:59:29 0:59:51
>> 0.62
>> kernel 5:01
>> 5:00 -0.33
>> eSDK 3:22
>> 3:19 -1.49
>>
>>
>>
>>
>> Fedora Test 2.2.2_rc1
>> 2.2.2_rc2 %
>> sato 1:01:55
>> 1:06:32 7.46
>> rootfs 2:49
>> 2:51 1.18
>> rmwork 0:59:52 1:01:32
>> 2.78
>> kernel 6:03
>> 6:47 12.12
>> eSDK 3:44
>> 3:54 4.46
>>
>>
>> ptest
>> There was an improvement on the pass rate in the following packages in
>> comparison with 2.2.2 rc1:
>> gdk-pixbuf with 5%
>> libxml2 with 4.20%
>> lttng-tools with 3.20%
>>
>> On the other side there was a regression on the pass rate in
>> comparison with 2.2.2 rc1:
>> acl with 6.80%
>> e2fsprogs with 100% regression
>> gawk with .10%
>> quilt with 1.80%
>> valgrind of 53.10%
>>
>>
>> Compliance
>> Results are yet to be documented by Windriver.
>>
>> ======= QA-Hints========
>>
>> There is only one concern on the eSDK component it cannot execute. QA
>> recommends an 2.2.2 rc3 to only verify the eSDK component.
>>
>> ======= Bugs ========
>>
>> New Bugs
>> -12075[1] [morty] testimage quemux86/qemux86-64 lsb
>> tcgetattr: Input/output error^M
>> -12080[2] [Morty][Test Case 1605]
>> TCTEMP_2.3_AUTO_sdkext_eSDK_devtool_build_make
>> -12073[3] selftest: setUpClass failed failing on ubuntu
>> 17.04 (oeqa.selftest.signing.Signing)
>> -12081[4] [morty] PTEST: Package Name flex, gzip, slang,
>> do not appear in 2.2.2 rc2
>>
>> Full Bug Report : https://wiki.yoctoproject.org/wiki/WW37_-_2017-09-1
>> 3_-_Full_Point_Release_Test_Cycle_2.2.2_rc2#Bugs_Found_during_QA_Test
>>
>>
>> ======== Links =========
>>
>> 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12075 [1]
>> 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12080 [2]
>> 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12073 [3]
>> 4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12081 [4]
>>
>>
>>
>> Regards
>> Libertad G.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-15 20:06 ` akuster808
@ 2017-09-15 20:14 ` Bystricky, Juro
2017-09-15 20:18 ` Cruz, Libertad
0 siblings, 1 reply; 8+ messages in thread
From: Bystricky, Juro @ 2017-09-15 20:14 UTC (permalink / raw)
To: akuster808, Cruz, Libertad, Richard Purdie,
yocto@yoctoproject.org, Jolley, Stephen K, Zhang, Jessica
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
I checked out 2.2_M3 2679a347c576f5411fbe802d2f6201c94036ecb2.
The recipes for gzip and slang do not contain any ptest code, so that explains why the tests are
missing.
However, flex recipe has a code for ptest. So there is a bit of mystery left.
________________________________________
From: akuster808 [akuster808@gmail.com]
Sent: Friday, September 15, 2017 1:06 PM
To: Bystricky, Juro; Cruz, Libertad; Richard Purdie; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica
Cc: Cruz Alcaraz, Juan M; Jordan, Robin L
Subject: Re: [yocto] QA cycle report for 2.2.2 rc2
On 09/15/2017 01:01 PM, Bystricky, Juro wrote:
> The commit is for 2.4 M3 (5f6945f5031e1a4ca116cc1eccf4c2f9dc228547)
> 2.4 does contain the ptests for gzip, slang and flex. (Although there are some other mysteries, like sed-ptest is missing)
> It would seem that 2.2 had never contained (or lost a while ago) those ptests.
Thanks for looking into this. Is there any action I need to follow up on?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: QA cycle report for 2.2.2 rc2
2017-09-15 20:14 ` Bystricky, Juro
@ 2017-09-15 20:18 ` Cruz, Libertad
0 siblings, 0 replies; 8+ messages in thread
From: Cruz, Libertad @ 2017-09-15 20:18 UTC (permalink / raw)
To: Bystricky, Juro, akuster808, Richard Purdie,
yocto@yoctoproject.org, Jolley, Stephen K, Zhang, Jessica
Cc: Cruz Alcaraz, Juan M, Jordan, Robin L
Ok, I will open a bug for QA to fix this ptest comparison issue, Bug 12094. I guess this ptest bug 12081 cannot be a blocker for the release.
Best Regards
Libertad G.
-----Original Message-----
From: Bystricky, Juro
Sent: Friday, September 15, 2017 3:14 PM
To: akuster808 <akuster808@gmail.com>; Cruz, Libertad <libertad.cruz@intel.com>; Richard Purdie <richard.purdie@linuxfoundation.org>; yocto@yoctoproject.org; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Cruz Alcaraz, Juan M <juan.m.cruz.alcaraz@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>
Subject: RE: [yocto] QA cycle report for 2.2.2 rc2
I checked out 2.2_M3 2679a347c576f5411fbe802d2f6201c94036ecb2.
The recipes for gzip and slang do not contain any ptest code, so that explains why the tests are missing.
However, flex recipe has a code for ptest. So there is a bit of mystery left.
________________________________________
From: akuster808 [akuster808@gmail.com]
Sent: Friday, September 15, 2017 1:06 PM
To: Bystricky, Juro; Cruz, Libertad; Richard Purdie; yocto@yoctoproject.org; Jolley, Stephen K; Zhang, Jessica
Cc: Cruz Alcaraz, Juan M; Jordan, Robin L
Subject: Re: [yocto] QA cycle report for 2.2.2 rc2
On 09/15/2017 01:01 PM, Bystricky, Juro wrote:
> The commit is for 2.4 M3 (5f6945f5031e1a4ca116cc1eccf4c2f9dc228547)
> 2.4 does contain the ptests for gzip, slang and flex. (Although there
> are some other mysteries, like sed-ptest is missing) It would seem that 2.2 had never contained (or lost a while ago) those ptests.
Thanks for looking into this. Is there any action I need to follow up on?
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-09-15 20:19 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-14 23:53 QA cycle report for 2.2.2 rc2 Cruz, Libertad
2017-09-15 16:50 ` Richard Purdie
2017-09-15 18:04 ` Bystricky, Juro
2017-09-15 19:50 ` Cruz, Libertad
2017-09-15 20:01 ` Bystricky, Juro
2017-09-15 20:06 ` akuster808
2017-09-15 20:14 ` Bystricky, Juro
2017-09-15 20:18 ` Cruz, Libertad
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.