QA cycle report for 2.3.3 RC1

QA cycle report for 2.3.3 RC1

[Cruz, Libertad]

Hello All,
Enjoy viewing the full Report for 2.3.3 RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1

======= Summary ========

The QA cycle for release 2.3.3 RC1 is complete.  There are 4 new bugs from which so far none of them are high. QA has two big concerns:

        1.  Performance report shows an increase of 30% build time on the eSDK in fedora 23, bug has not been created until further investigation with setup outside of the GDC environment. 
        2.  ptest comparison script is giving wrong results, bug 12441 was opened. 


-----

ptest

Package slang show 0% pass rate in 2.3.3 rc1, the results need to be validated, because bug 12441 impacts the comparison script.

-----

Toaster
Exploratory testing was executed in this release.

-----
Performance

The performance results showed regression on some of the measurements compared to 2.3.2rc1 info showed below, the most important one is on building eSDK that is taking around 30% of increase on fedora machine.


Ubuntu 15                          Test       2.3.2 rc1              2.3.3 rc1              %
                                             sato       1:12:11                1:16:29                +5.96
                                             rootfs    2:49                      2:53                      +2.37
                                             rmwork 1:04:15                1:06:47                +3.94
                                             kernel    5:21                      5:25                      +1.25
                                             eSDK      2:46                      2:46                      0.00
                                                                           
                                                                           
Fedora 23                           Test       2.3.2 rc1              2.3.3 rc1              %
                                             sato       1:14:51                1:21:20                +8.66
                                             rootfs    2:45                      2:44                      -0.61
                                             rmwork 1:07:47                1:14:54                +10.50
                                             kernel    5:56                      6:04                      +2.25
                                             eSDK      3:06                      4:12                      +35.48


https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1#Performance_Charts

======= QA-Hints========


The build is stable enough to be released, provided the following results are checked and investigated:

    1. Performance needs to be executed in an environment outside of GDC in another fedora 23, volunteers?
    2. QA asks ptest for slang package be checked by development to verify that it is executing.

======= Bugs ========

       New Bugs
            -12440[1] test_qemu_efi (oeqa.selftest.wic.Wic) fails on Fedora26
            -12441[2] ptest comparison script is not showing correct results
            -12436[3] [Test Case 1958] Project names cannot be blank
            -12437[4] [Test Case 1958] Toaster exploratory testingin the test run.

       Not new M+ bugs
            -12022[5] signing.Signing.test_signing_packages hangs on ubuntu1704.yocto.io

Full Bug Report:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1#Bugs_Found_during_QA_Test



======== Links =========
    1.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12440[1] 

    2.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12441[2] 

    3.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12436[3] 

    4.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12437[4] 

    5.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12022 [5] 
   

Regards
Libertad G.

Re: QA cycle report for 2.3.3 RC1

[Richard Purdie]

On Wed, 2017-12-20 at 20:25 -0800, Cruz, Libertad wrote:
> Hello All,
> Enjoy viewing the full Report for 2.3.3 RC1:  https://wiki.yoctoproje
> ct.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1
> 
> ======= Summary ========
> 
> The QA cycle for release 2.3.3 RC1 is complete.  There are 4 new bugs
> from which so far none of them are high. QA has two big concerns:
> 
>         1.  Performance report shows an increase of 30% build time on
> the eSDK in fedora 23, bug has not been created until further
> investigation with setup outside of the GDC environment. 

Could you confirm when the performance test machines were last
rebooted? We've had issues before where the benchmarks drift with
machine uptime. If the've been running for more than about 3 weeks I'd
like to run the benchmarks again after a reboot.

Thanks!

Richard

Re: QA cycle report for 2.3.3 RC1

[Jose Perez Carranza]

On 12/21/2017 08:43 AM, Richard Purdie wrote:
> On Wed, 2017-12-20 at 20:25 -0800, Cruz, Libertad wrote:
>> Hello All,
>> Enjoy viewing the full Report for 2.3.3 RC1:  https://wiki.yoctoproje
>> ct.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1
>>
>> ======= Summary ========
>>
>> The QA cycle for release 2.3.3 RC1 is complete.  There are 4 new bugs
>> from which so far none of them are high. QA has two big concerns:
>>
>>          1.  Performance report shows an increase of 30% build time on
>> the eSDK in fedora 23, bug has not been created until further
>> investigation with setup outside of the GDC environment.
> Could you confirm when the performance test machines were last
> rebooted? We've had issues before where the benchmarks drift with
> machine uptime. If the've been running for more than about 3 weeks I'd
> like to run the benchmarks again after a reboot.
It was last week 12/12

-$ who -b
          system boot  2017-12-12 09:18

-$ uptime -s
          2017-12-12 09:19:28

> Thanks!
>
> Richard

-- 
Saludos
José

FW: QA cycle report for 2.3.3 RC1

[Cruz, Libertad]

Machine was rebooted

last week 12/12

-$ who -b
          system boot  2017-12-12 09:18

-$ uptime -s
          2017-12-12 09:19:28



-----Original Message-----
From: Perez Carranza, Jose 
Sent: Thursday, December 21, 2017 8:57 AM
To: Purdie, Richard <richard.purdie@intel.com>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Lock, Joshua G <joshua.g.lock@intel.com>; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Sangal, Apoorv <apoorv.sangal@intel.com>; Yeoh, Ee Peng <ee.peng.yeoh@intel.com>
Subject: Re: QA cycle report for 2.3.3 RC1


On 12/21/2017 08:43 AM, Richard Purdie wrote:
> On Wed, 2017-12-20 at 20:25 -0800, Cruz, Libertad wrote:
>> Hello All,
>> Enjoy viewing the full Report for 2.3.3 RC1:  https://wiki.yoctoproje
>> ct.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1
>>
>> ======= Summary ========
>>
>> The QA cycle for release 2.3.3 RC1 is complete.  There are 4 new bugs 
>> from which so far none of them are high. QA has two big concerns:
>>
>>          1.  Performance report shows an increase of 30% build time 
>> on the eSDK in fedora 23, bug has not been created until further 
>> investigation with setup outside of the GDC environment.
> Could you confirm when the performance test machines were last 
> rebooted? We've had issues before where the benchmarks drift with 
> machine uptime. If the've been running for more than about 3 weeks I'd 
> like to run the benchmarks again after a reboot.
It was last week 12/12

-$ who -b
          system boot  2017-12-12 09:18

-$ uptime -s
          2017-12-12 09:19:28

> Thanks!
>
> Richard

--
Saludos
José