QA cycle report for 2.4.1 RC1

QA cycle report for 2.4.1 RC1

[Yeoh, Ee Peng]

Hello All,
Enjoy viewing the full Report for 2.4.1 RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-_Full_Test_Cycle_-_2.4.1_rc1

======= Summary ========

The QA cycle for release 2.4.1 RC1 is complete.  There are 5 new bugs from which so far none of them are high. QA has 1 concern on the performance testing.

Performance report shows an increase of 23% build time on the eSDK in fedora 23, bug has not been created until further investigation with setup outside of the GDC environment. 

https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-_Full_Test_Cycle_-_2.4.1_rc1#Build_Performance_Test

======= QA-Hints========

The build is stable enough to be released, provided the performance issue is investigated and root-cause identified.

======= Bugs ========

       New Bugs
            -12444 [1] [Test Case 1542] test_2_logrotate fails in MinnowMax with core-image-lsb-sdk-genericx86-64 image
            -12417 [2] Eclipse installer steps only apply to latest Eclipse release
            -12416 [3] autobuilder.yocto.io directory index does not display the full long filename
            -12443 [4] [Test Case 312,313] Bitbake failed after using yocto-kernel add patch
            -12445 [5] [Test Case 309,310,311] Bitbake fails after add layer with yocto-bsp create

Full Bug Report:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-_Full_Test_Cycle_-_2.4.1_rc1#Bugs_Found_during_QA_Test


======== Links =========
    1.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12444 [1] 

    2.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12417 [2] 

    3.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12416 [3] 

    4.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12443 [4]

    5.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12445 [5] 
   

Regards
Ee Peng 

Re: QA cycle report for 2.4.1 RC1

[Richard Purdie]

Hi,

On Fri, 2017-12-22 at 01:40 -0800, Yeoh, Ee Peng wrote:
> Enjoy viewing the full Report for 2.4.1
> RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-
> _Full_Test_Cycle_-_2.4.1_rc1
> 
> ======= Summary ========
> 
> The QA cycle for release 2.4.1 RC1 is complete.  There are 5 new bugs
> from which so far none of them are high. QA has 1 concern on the
> performance testing.
> 
> Performance report shows an increase of 23% build time on the eSDK in
> fedora 23, bug has not been created until further investigation with
> setup outside of the GDC environment. 

Was there any further update on this?

It may be worth running the test for 2.4.0 and checking that you get
the same result as we had at time of release, just to double check
there isn't some other contributing cause here...

Cheers,

Richard

Re: QA cycle report for 2.4.1 RC1

[Yeoh, Ee Peng]

[-- Attachment #1.1: Type: text/plain, Size: 1928 bytes --]

Hi Richard,



We had just configured Fedora Performance Machine to run 2.4.rc3 after we had collected enough performance data for 2.5 M1 rc1.

Please give us 2 days to collect 10 data points to verify this issue.

[cid:image003.jpg@01D3847B.47E721D0]



Hi Jose,



I had configured ypperf01 (Fedora Performance Machine) to run 2.4.rc3.

Please let me know if anything that concern you.

[cid:image004.jpg@01D3847B.47E721D0]



Thank you very much!



Thanks,

Yeoh Ee Peng



-----Original Message-----

From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
Sent: Tuesday, January 2, 2018 7:27 PM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Lock, Joshua G <joshua.g.lock@intel.com>; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: Re: QA cycle report for 2.4.1 RC1



Hi,



On Fri, 2017-12-22 at 01:40 -0800, Yeoh, Ee Peng wrote:

> Enjoy viewing the full Report for 2.4.1

> RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-

> _Full_Test_Cycle_-_2.4.1_rc1

>

> ======= Summary ========

>

> The QA cycle for release 2.4.1 RC1 is complete.  There are 5 new bugs

> from which so far none of them are high. QA has 1 concern on the

> performance testing.

>

> Performance report shows an increase of 23% build time on the eSDK in

> fedora 23, bug has not been created until further investigation with

> setup outside of the GDC environment.



Was there any further update on this?



It may be worth running the test for 2.4.0 and checking that you get the same result as we had at time of release, just to double check there isn't some other contributing cause here...



Cheers,



Richard

[-- Attachment #1.2: Type: text/html, Size: 6105 bytes --]

[-- Attachment #2: image003.jpg --]
[-- Type: image/jpeg, Size: 15425 bytes --]

[-- Attachment #3: image004.jpg --]
[-- Type: image/jpeg, Size: 24642 bytes --]

Re: QA cycle report for 2.4.1 RC1

[Perez Carranza, Jose]

[-- Attachment #1.1: Type: text/plain, Size: 2730 bytes --]

Hi Ee Peng

Go ahead,  I don’t have any concern.

Regards,
José

From: Yeoh, Ee Peng
Sent: Tuesday, January 2, 2018 8:12 PM
To: Richard Purdie <richard.purdie@linuxfoundation.org>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Lock, Joshua G <joshua.g.lock@intel.com>; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: RE: QA cycle report for 2.4.1 RC1


Hi Richard,



We had just configured Fedora Performance Machine to run 2.4.rc3 after we had collected enough performance data for 2.5 M1 rc1.

Please give us 2 days to collect 10 data points to verify this issue.

[cid:image002.jpg@01D38466.1275D880]



Hi Jose,



I had configured ypperf01 (Fedora Performance Machine) to run 2.4.rc3.

Please let me know if anything that concern you.

[cid:image004.jpg@01D38466.1275D880]



Thank you very much!



Thanks,

Yeoh Ee Peng



-----Original Message-----

From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
Sent: Tuesday, January 2, 2018 7:27 PM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com<mailto:ee.peng.yeoh@intel.com>>; Cruz, Libertad <libertad.cruz@intel.com<mailto:libertad.cruz@intel.com>>; yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>; Lock, Joshua G <joshua.g.lock@intel.com<mailto:joshua.g.lock@intel.com>>; Jolley, Stephen K <stephen.k.jolley@intel.com<mailto:stephen.k.jolley@intel.com>>; Zhang, Jessica <jessica.zhang@intel.com<mailto:jessica.zhang@intel.com>>
Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com<mailto:jose.perez.carranza@intel.com>>; Sangal, Apoorv <apoorv.sangal@intel.com<mailto:apoorv.sangal@intel.com>>
Subject: Re: QA cycle report for 2.4.1 RC1



Hi,



On Fri, 2017-12-22 at 01:40 -0800, Yeoh, Ee Peng wrote:

> Enjoy viewing the full Report for 2.4.1

> RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-

> _Full_Test_Cycle_-_2.4.1_rc1

>

> ======= Summary ========

>

> The QA cycle for release 2.4.1 RC1 is complete.  There are 5 new bugs

> from which so far none of them are high. QA has 1 concern on the

> performance testing.

>

> Performance report shows an increase of 23% build time on the eSDK in

> fedora 23, bug has not been created until further investigation with

> setup outside of the GDC environment.



Was there any further update on this?



It may be worth running the test for 2.4.0 and checking that you get the same result as we had at time of release, just to double check there isn't some other contributing cause here...



Cheers,



Richard

[-- Attachment #1.2: Type: text/html, Size: 8079 bytes --]

[-- Attachment #2: image002.jpg --]
[-- Type: image/jpeg, Size: 15405 bytes --]

[-- Attachment #3: image004.jpg --]
[-- Type: image/jpeg, Size: 24627 bytes --]

Re: QA cycle report for 2.4.1 RC1

[Yeoh, Ee Peng]

[-- Attachment #1.1: Type: text/plain, Size: 2712 bytes --]

Hi Richard,



We had configured GDC Fedora Performance Machine to run 2.4 M4 rc3 [1].

Based on the new performance data we collected, the average time for eSDK deploy is 237 seconds (3 min 57 sec). In comparison, data collected WW43 2017, the average time for eSDK deploy was 161 seconds (2 min 41 sec).



In summary, while running the same commit, 2.4 M4 rc3, the eSDK time is now significantly higher (47% higher). This clearly show that the performance machine was the root cause of the longer eSDK time during recent QA for 2.4.1.rc1 [2] but we have not identify why the performance machine was contributing to this trouble.



Any suggestion on what shall be our next step?



Hi Jose,



Any idea what could be the reason why performance machine take longer time to run eSDK deploy? How about restart the Fedora Performance Machine?



Thank you very much for your attention!



Thanks,

Yeoh Ee Peng



eSDK Deploy Time – Run on WW01 2018

[cid:image001.jpg@01D38649.9DBA1930]



eSDK Deploy Time – Run on WW43 2017

[cid:image002.jpg@01D38649.9DBA1930]





[1] https://wiki.yoctoproject.org/wiki/WW43_-_2017-10-23_-_Full_Test_Cycle_2.4_RC3

[2] https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-_Full_Test_Cycle_-_2.4.1_rc1

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
Sent: Tuesday, January 2, 2018 7:27 PM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; Cruz, Libertad <libertad.cruz@intel.com>; yocto@yoctoproject.org; Lock, Joshua G <joshua.g.lock@intel.com>; Jolley, Stephen K <stephen.k.jolley@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>
Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com>; Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: Re: QA cycle report for 2.4.1 RC1



Hi,



On Fri, 2017-12-22 at 01:40 -0800, Yeoh, Ee Peng wrote:

> Enjoy viewing the full Report for 2.4.1

> RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-

> _Full_Test_Cycle_-_2.4.1_rc1

>

> ======= Summary ========

>

> The QA cycle for release 2.4.1 RC1 is complete.  There are 5 new bugs

> from which so far none of them are high. QA has 1 concern on the

> performance testing.

>

> Performance report shows an increase of 23% build time on the eSDK in

> fedora 23, bug has not been created until further investigation with

> setup outside of the GDC environment.



Was there any further update on this?



It may be worth running the test for 2.4.0 and checking that you get the same result as we had at time of release, just to double check there isn't some other contributing cause here...



Cheers,



Richard

[-- Attachment #1.2: Type: text/html, Size: 7424 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 17944 bytes --]

[-- Attachment #3: image002.jpg --]
[-- Type: image/jpeg, Size: 13673 bytes --]